Determining root-cause of failures based on machine-generated textual data

ABSTRACT

A method and system for determining root-causes of incidences using machine-generated textual data. The method comprises receiving machine-generated textual data from at least one data source; classifying the received machine-generated textual data into at least one statistical metric; processing the statistical metric to recognize a plurality of incidence patterns; correlating the plurality of incidence patterns to identify at least a root-cause of an incidence that occurred in a monitored environment; and generating an alert indicating at least the identified root-cause.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/499,060, filed on Apr. 27, 2017; which is a continuation-in-part ofU.S. patent application Ser. No. 15/228,272 filed on Aug. 4, 2016, thecontents of which are hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure generally relates to root-cause analysis ofincidences using machine-generated data.

BACKGROUND

The amount of data generated by various machines (e.g., appliances,servers, software tools, etc.) connected in an organization is enormous.The machine-generated data may be in a structured textual formant, anunstructured textual format, or a combination thereof. Examples for suchmachine-generated textual data include logs, metrics, configurationfiles, messages, spreadsheets, events, alerts, sensory signals, auditrecords, database tables, and so on. The various machines in anenterprise are typically from multiple different vendors and, thus, evenif the data from each vendor is in a structured format, that data is nottypically unified across different vendors. Additionally,machine-generated textual data is not in a natural language that can beread and understood by humans, as machines are currently not adapted torecognized such data.

The vast amount of machine-generated textual data requires informationtechnology (IT) personnel to effectively review, analyzed and responseto countless unwanted emails, messages, notifications, and the like toidentify a specific malfunction. The ability of a person (e.g., an ITadministrator) to react to such high volumes of data is limited by therate of processing of the person. Further, the high volume of datadecreases productivity and delays detection of critical issues, as notall data can be processed by the person.

Moreover, a user that needs to process such large volumes of data maywish to gain visibility as to the performance of the entire IT systemsin the enterprises and determine a root-cause for reported malfunction.To determine the causality between reported alerts data received fromdifferent domains (e.g., network, infrastructure, and application)should be processed. Each such domain has its own domain-expert. Thus,the challenge of determining the root-cause of each incident isamplified. For example, the machine-generated textual data may includereadings indicative of a high-CPU utilization and security logsindicative of new viruses. Currently, IT personnel have no effective wayto determine any causality between these reported inputs.

Existing solutions cannot resolve the deficiencies noted above, as suchsolutions operate in silos. That is, the creation of machine-generatedtextual data and reading of such data are performed by differentsolutions (components), which are not necessarily developed by the samevendors. Furthermore, some existing solutions for digitalevents-ingestion merely aggregate machine-generated data and providesearch capabilities across the aggregated data. Other solutions arelimited to processing a specific set of textual data generated by commontools. However, such solutions typically do not cover the entirespectrum of machines installed in an organization and are not adapted tocover the entire set of logs, events, and other data generated by themachines. Therefore, meaningful and important information may not bedetected or otherwise analyzed by such solutions.

Existing solutions fail to detect a root-cause and its symptoms of amalfunction. A malfunction can be a system error or failure, anapplication error, and the like. This deficiency is largely due to atleast the following challenges: the need to query multiple data-sourcesstoring data in different structures at the same time; the structure ofmachine-generated data not being standardized; the data being formattedwith the intention that the data is to be ingested by a human ratherthan a computer; the machine-generated data including a mixture of theoriginal events, wrapped with unrelated additional information (e.g.,Syslog transport headers added by relay servers); and the same databeing serialized in several formats (e.g. JSON, XML).

As a result of the deficiencies of existing solutions, machine-generatedtextual data is often analyzed by humans. Of course, any manual analysisis prolonged, requires human resources, and affects the overallperformance of the enterprise. A major drawback of this approach is thatthe amount of data that can be processed by users such as IT personnelis limited by restraints on human resources. Due to the size, variety,retention, and dynamic nature of machine-generated data that continuesto grow, a manual approach for solving the above-noted tasks isinefficient.

In addition to the above limitations, current solutions generate alertsbased on rules or thresholds set by the IT personnel. During typicaloperation, such rules and thresholds often result in a flood of alertsof different sensitivities. As a result, many alerts are ignored oroverlooked. Further, the generated alerts typically do not includeinformation on the root-cause of the problem that triggered an alert orrecommendations how to solve the detected indecent.

It would therefore be advantageous to provide a solution that wouldovercome the deficiencies of the prior art.

SUMMARY

A summary of several example embodiments of the disclosure follows. Thissummary is provided for the convenience of the reader to provide a basicunderstanding of such embodiments and does not wholly define the breadthof the disclosure. This summary is not an extensive overview of allcontemplated embodiments, and is intended to neither identify key orcritical elements of all embodiments nor to delineate the scope of anyor all aspects. Its sole purpose is to present some concepts of one ormore embodiments in a simplified form as a prelude to the more detaileddescription that is presented later. For convenience, the term “someembodiments” may be used herein to refer to a single embodiment ormultiple embodiments of the disclosure.

Some embodiments disclosed herein include a method for determiningroot-causes of incidences using machine-generated textual data. Themethod comprises receiving machine-generated textual data from at leastone data source; classifying the received machine-generated textual datainto at least one statistical metric; processing the statistical metricto recognize a plurality of incidence patterns; correlating theplurality of incidence patterns to identify at least a root-cause of anincidence that occurred in a monitored environment; and generating analert indicating at least the identified root-cause.

Some embodiments disclosed herein include a system for determiningroot-causes of incidences using machine-generated textual data. Thesystem comprises a processing circuit; a memory communicativelyconnected to the processing circuit, wherein the memory containsinstructions that, when executed by the processing element, configurethe processing circuit to: receive machine-generated textual data fromat least one data source; classify the received machine-generatedtextual data into at least one statistical metric; process thestatistical metric to recognize a plurality of incidence patterns;correlate the plurality of incidence patterns to identify at least aroot-cause of an incidence that occurred in a monitored environment; andgenerate an alert indicating at least the identified root-cause.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter disclosed herein is particularly pointed out anddistinctly claimed in the claims at the conclusion of the specification.The foregoing and other objects, features, and advantages of thedisclosed embodiments will be apparent from the following detaileddescription taken in conjunction with the accompanying drawings.

FIG. 1 is a network diagram utilized to describe the various disclosedembodiments.

FIG. 2 shows a simulation illustrating a time-proximity basedcorrelation type.

FIG. 3 shows a simulation illustrating an order-based correlation type.

FIG. 4 is a schematic diagram illustrating component-based correlationtype according to an embodiment.

FIG. 5 is a diagram illustrating an IT infrastructure utilized todescribe the component-based correlation type.

FIG. 6 is a flowchart illustrating a method for root-cause analysis ofincidence using machine-generated textual data according to anembodiment.

FIG. 7 is a block diagram of a server configured to carry out thevarious disclosed embodiments.

DETAILED DESCRIPTION

It is important to note that the embodiments disclosed herein are onlyexamples of the many advantageous uses of the innovative teachingsherein. In general, statements made in the specification of the presentapplication do not necessarily limit any of the various claimedembodiments. Moreover, some statements may apply to some inventivefeatures but not to others. In general, unless otherwise indicated,singular elements may be in plural and vice versa with no loss ofgenerality. In the drawings, like numerals refer to like parts throughseveral views.

Some example embodiments disclosed herein include a method and systemfor detecting the root-cause of incidences based on machine-generatedtextual data provided by different sources. Incidences may includefailures, malfunctions, and the like of any resource in a monitoredenvironment. As will be discussed in more detail below, in anembodiment, the root-cause analysis is performed by aggregating andcorrelating potential incidence patterns. In certain embodiments, thedetermined root-cause can be utilized to reduce the number of alertstriggered in response to detection of incidences, and for prioritizationof such alerts. In an embodiment, the machine-generated textual data iscollected from one or more data sources. The collected data isreconstructed and classified into different statistical metrics. Thestatistical metrics may include, for example, a gauge, a meter, and ahistogram.

FIG. 1 shows an example network diagram 100 utilized to describe thevarious disclosed embodiments. The network diagram 100 includes a clientdevice 110, a network 120, a server 130, and a plurality of data sources140-1 through 140-n (hereinafter referred to individually as a datasource 140 and collectively as data sources 140, merely for simplicitypurposes), In some embodiments, the network diagram 100 further includesa database 150 communicatively connected to the network 120 and utilizedto store machine-generated textual data, events processed based onmachine-generated textual data, and the classification results.

The client device 110 may be operated by a user (e.g., a systemadministrator) to control the operation of the server 130, view alerts,detected incidences, and so on. The network 120 may be, but is notlimited to, a local area network (LAN), a wide area network (WAN), theInternet, a wired network, a wireless network, similar networks, and thelike, as well as any combination thereof.

Each of the data sources 140 generates machine-generated textual data.The data sources 140 may be different machines, systems, or softwaretools operable in an organization and configured to monitor, control,and report on issues related to, for example, computing infrastructureof an organization. Examples for the data sources 140 include anyIT-based device, such as routers, network appliances, applicationservers, database servers, sensors, and the like. In general, anyvirtual or physical computing resource adapted to generate textual datacan serve as a data source 140.

The machine-generated textual data generated by the data sources 140 mayinclude, for example, application logs, configuration files, messages,spreadsheets, events, alerts, sensory signals, audit records, and so on.It should be noted that the data sources 140 are different from eachother and, thus, the data provided by each source may be of a differentformat, structure, or both. Furthermore, some of the data sources 140may output structured data while others may output unstructured data.The machine-generated textual data provided by the data sources 140 maybe standardized or not-standardized.

The machine-generated textual data may be encapsulated in CSV files,JSON files, XML files, plain text files, and so on. Such files can bepulled by the server 130 (from the sources 140), pushed to the server130, uploaded to the server 130, received at the server 130 throughother methods or channels, or a combination thereof. Typically, JSON andXML files are streamed in real-time, while CVS files are uploaded inbatches.

In an embodiment, the server 130 is configured to detect the root-causeof incidences. To this end, the server 130 is configured to classify thereceived machine-generated textual data into a set of statisticalmetrics. Types of the statistical metrics include, but are not limitedto, a histogram, a meter, a gauge, and the like. For example, CPUutilization and page-views may be classified as gauge metrics, purchasesmay be classified as a histogram metric, and message-clusters andhostnames may be classified as meter metrics.

In addition, the server 130 is configured to output metadata associatedwith each event in the machine-generated textual data and a schema thatcan be used to structure the event. The schema allows for performance ofvarious operations on events, such as querying, filtering, manipulating,calculating statistical attributes, or otherwise handling the event orparts of the event.

The server 130 is further configured to process the various portions ofan event to identify contextual tokens. Typically, contextual tokens areidentified in messages included in the event. A message includesfree-text that cannot be categorized as a property and has a length(e.g., number of characters) above a predefined threshold.

The metric utilized to classify events is determined for clusters ofevents, tokens, key-value pairs, and properties. In an embodiment, thedetermination of which metric type to associate with each such elementis based on the element's type, context, or a combination thereof. Thatis, for each identified element (e.g., elements such as clusters,tokens, key-value pairs, and properties), it is determined whether theelement's value can be statistically measured as any or all of a valuedistribution (histogram), an appearance rate (meter), or a value range(gauge). In an embodiment, the context of an element can be utilized todetermine its statistical value distribution. A metric is a statisticalvalue distribution.

An example classification process is described in more detail in theco-pending U.S. patent application Ser. No. 15/228,272 (hereinafter the'272 Application), assigned to the common assignee. which is herebyincorporated herein by reference.

In an embodiment, the server 130 is configured to recognize incidencepatterns by processing the statistical metrics. In a further embodiment,the server 130 is further configured to apply past behavior of eachincidence pattern to predict its future behavior. The incidence patternsmay include, but are not limited to, new behavior, anomalous behavior,changes in routine operational, ongoing trends, and so on. The incidencepattern recognition may be performed using one or more techniquesincluding, but not limited to, statistical methods (e.g., Grubbs andAR/MA utilized to detect local outliers over different timespans),frequencies analysis to detect deviations from expected seasonalbehavior, Hidden Markov Models to pinpoint a transition between normaland abnormal states. Other techniques may include Kolmogorov-Smimov andU-Test used to identify changes in the basic distribution over time,when such changes are relevant.

In an embodiment, the type of technique utilized to recognize thepattern is based on the metric being processed. In another embodiment,every recognized pattern is further analyzed to determine if an alertshould be generated. In an example, the analysis is performed using anAbaBoost technique which results in a yes/no decision.

The recognized incidence patterns are processed to determine aroot-cause of an incidence. In an embodiment, a set of “meaningful”incidence patterns from the recognized incidence patterns is processed.To this end, the server 130 is configured to determine which of therecognized incidence patterns are meaningful. In an embodiment,meaningful patterns are determined by considering the amplitude of apattern, a frequency of a pattern, a similarity of a pattern topreviously detected patterns, a number of same or similar recognizedpatterns, or any combination thereof. An amplitude of a pattern may be,for example, an amplitude of a corresponding anomaly. A frequency of apattern may be, for example, the number of times the same anomaly wasobserved during of a predefined time interval. The selection ofmeaningful incidence patterns improves the processing time and thequality of the analysis.

In most cases, a single root problem is caused by incidence patterns(e.g., anomalous behavior) of different related or unrelated events.This results in numerous alerts being triggered. Thus, detecting theroot-cause of an incident would allow for reducing the number of alertsbeing reported to the user. Further, detecting the root-cause of anincident may allow for grouping of different alerts that were generatedsimultaneously or near simultaneously. The grouping may be based on theurgency and/or importance of the different alerts. Such grouping wouldincrease the coverage of the entity being tracked and/or monitored.

According to the disclosed embodiments, the server 130 is furtherconfigured to determine the root-cause of an incidence by aggregatingand correlating at least two recognized incidence patterns. Theaggregation may be over a predefined time-period or when there is asufficient number of recognized patterns.

In one embodiment, the correlation is time-proximity based. That is, twoor more incidence patterns resulting from events that occurred at thesame or substantially the same time are correlated. In this embodiment,the root-cause is determined to be an incidence observed by an incidencepattern that occurred before other incidence patterns being correlated.

This correlation type is further demonstrated in the example simulationshown in FIG. 2. Graphs 210 and 220 show two incidence patterns 201 and202, respectively. The pattern 201 demonstrates a normal behavior fromT₀ to T₁ and after T₃, and abnormal behavior between T₁ and T₃. Thepattern 202 demonstrates a normal behavior from T₀ to T₂ and after T₄,and abnormal behavior between T₂ and T₄. In this embodiment, theincidence indicated by the pattern 201 is the root-cause of theincidence indicated by the pattern 202.

As an example, a first incidence pattern occurs due to a gauge metricdemonstrating an anomaly resulting from a high CPU utilization. Thetimestamp associated with the logs generated in response to the high CPUutilization is 00:05. A second incidence pattern occurs due to a metermetric demonstrating an anomaly resulting from a high number of securitylogs indicative of a newly detected malware. The timestamps associatedwith the events generated in response to security logs is 00:00-00:12.As the time proximity of these incidence patterns are relatively close,they are determined to be time correlated. The correlation of thepatterns would determine that the root-cause of the high CPU utilizationis the new malware. It should be noted that the incidence patterns arecorrelated across many applications and systems, and that the onlyselection factor in this embodiment is time-based.

In another embodiment, the incidence patterns are correlated based on acertain order. The order may be based on time, severity, or both. Thiscorrelation type attempts to identify at least one potential incidencethat caused the other potential incidences. This type of correlation isfurther demonstrated in the example simulation shown in FIG. 3.

The graphs 310, 320, and 330 represent different incidence patterns 301,302, and 303, respectively. Each of the graphs 310, 320, and 330 furtherdemonstrates the time and severity of each incidence pattern. Theseverity may be, for example, an amplitude, a frequency, and the like.In this embodiment, the server 230 is configured to detect at least oneincidence pattern that causes other incidences to “break”. That is, anincidence represented by an incidence pattern trended as an increased,for example, in severity, but has not yet crossed the severitythreshold.

As shown in FIG. 3, at T₀, the incidence patterns 301, 302, and 302 arebelow their respective thresholds TH-A, TH-B, and TH-C. The severity ofthe pattern 301 gradually increases from T₁ to T₂. until the thresholdlevel TH-A is crossed. At time T₂, all other incidence patterns 302 and303 cross their respective threshold levels TH-B and TH-C, respectively.Thus, the incidence represented by the incidence pattern 301 broke theincidence patterns 302 and 303. As such, the incidence observed by thepattern 301 is the root-cause of incidences observed by patterns 302 and303.

As an example, a first incidence pattern occurs due to a storage devicehaving a disk malfunction which does not allow any data writes to thedisk. Due to such a malfunction, the storage device would attempt torewrite (reperform) any write request. As a result, the operation speedof the storage device would be reduced until the device becomesunresponsive. When the storage device ceases operation, any servicewriting to the storage device would also be unresponsive. Thus, thealert would be generated for each unresponsive service.

In this example, the incidence patterns are meter metrics resulting fromerror messages generated by the storage device and each service writingto the device. The incidence patterns all become broken when the storagedevice ceases operation. Thus, the correlation results in determiningthe root-cause of the unresponsive service to be the storage device witha malfunction disk.

In another embodiment, the root-cause can be determined when there is nocertain order or no commonality across the observed potentialincidences. In this embodiment, the root-cause is determined based onthe potential incidences correlated across different components. Acomponent may be, for example, any group of services, applications,devices, or hardware elements serving the same function. This type ofcorrelation will be discussed with reference to FIG. 4.

FIG. 4 is an example simulation showing 3 different components 410, 420,and 430, each of which represent similar elements 411, 422, and 433,respectively. The incidence patterns of the elements 422 and 433indicate an incidence (e.g., a failure). In the component 410, theincidence pattern of only element 411-X indicates an incidence, whilethe other elements are non-incidental. All incidence patterns of allelements are generated based on time proximity. In this embodiment, theroot-cause is determined to be the element in a component for which theother elements are not broken. In the example simulation shown in FIG.4, the element 411-X is the root-cause.

The component-based correlation type is further discussed in FIG. 5,which shows an IT infrastructure (e.g., a datacenter) 500 including apair of load balancers 510-1 and 510-2 communicatively connected to aplurality of application (App) servers 520-1 through 520-4. The loadbalancers 510 and 511 distribute incoming requests (e.g., HTTP requests)to the application servers 520-1 through 520-4. The load balancers 510-1and 510-2 are grouped as component A, while the application servers520-1 through 520-4 are grouped as component B.

In this example, the load balancer 510-1 is erroneously configured todistribute all requests to the application servers 520-1 and 520-4. Theload balancer 510-2 is properly configured. As a result, the incidencepattern of the load balancer 510-2 is a meter metric due to a badconfiguration error message.

The incidence patterns of the application servers 520-1 through 520-4are gauge metrics indicating high load for the servers 520-1 and 520-4and low CPU utilization for the servers 520-2 and 520-3. It should benoted that there is no specific order to the generation of incidencepatterns and not all patterns resulted from the same metric type.Further, the component A does not demonstrate a potential incidenceacross all of its elements.

In an embodiment, the incidence patterns are detected approximately atthe same time, but not necessarily in any specific order. The root-causeis determined to be in a component if at least some elements of thecomponent report a problem. In this example, the root-cause of theproblem in the application severs 520 is the bad configuration in theload balancer 510-1.

It should be noted that, without correlating the incidence patterns assuggested herein, an IT person would typically look for problem at theapplication servers 520. The user will report problems on theapplication servers 520 (e.g., cannot access a website or make an onlinepurchase). IT personnel examining logs of the application servers 520cannot determine if the failure is due to bad configuration of the loadbalancers 510.

In another embodiment, the detection of a root-cause of is based ontracing certain entities identified in at least one incidence pattern ofa detected incidence (anomaly). The identified entities containmeaningful information that would allow detection of the root-cause.Examples for identifiers may include, a machine name or identifierreported a failure, a user identifier, a reason for the failure,timestamp, and so on. The machine identifier may be, for example, ahostname, an IP address, and the like.

The incidence patterns generated prior to the detection of the incidenceare scanned to detect entities that are the same or similar to theidentified entities. In an embodiment, a predefined time interval isused for the scanning, e.g., only incidence patterns including atimestamp of an hour prior to the detection of the incidence. Anyincidence patterns including entities matching the identified entitiesare correlated to determine the root-cause. In another embodiment, suchincidence patterns are displayed to the user on a timeline of theiroccurrence.

It should be appreciated that as incidence patterns are generated bydifference applications and/or sources, the disclosed embodiments allowdetection of a cause of an incidence across applications. For example, acause for a failure of an application may be a firewall that blocked aspecific IP address.

In another embodiment, the detection of a root-cause of a singleincidence pattern is detected. That is, the root-cause is determinedwithout correlation of two or more incidence patterns. In thisembodiment, upon detection of an incidence, machine-generated textualdata associated with the respective incidence pattern are gathered. Inan embodiment, machine-generated textual data received during a pastpredefined time window is gathered. For example, all application logsgenerated in the last hour since the detection of the incidence aregathered. The gathered machine-generated textual data may be classifiedor unclassified.

The gathered machine-generated textual data is analyzed to detect one ormore entities that may cause the incidence. The values of one, some orall of the entities are extracted from the gathered textual data andcompared to their respective historic values. A historic valuerepresents at least normal behavior of an entity. Any entity thatdeviates from its respective value may indicate the root cause of theincidence. The entities to be analyzed may be predetermined.

As an example, a failure in a number of servers installed in a rackcabinet is detected at 22:27. The sensory signals including readings ofthe temperature in the rack cabinet since 21:27 are gathered andcompared to their respective historic values. Higher readings in thetemperature since 21:27 would indicate that the root cause is a rack'sfan failure. Returning to FIG. 1, in an embodiment, the server 130 isconfigured to generate an alert indicating the determined root-cause. Inanother embodiment, the server 130 is configured to group togetheralerts which have a common cause into one incident. In yet anotherembodiment, the server 130 is further configured to report any alertremaining after the root-cause analysis with additional informationrelated to the associated incidence or incidences. That is, an alertwould indicate the detected root-cause and would be reported with theassociated incidence(s). This allows the user to drill down and betterunderstand the problem potential solutions.

It should be noted that all incidence patterns are generated frommachine-generated textual data which are automatically processed andclassified into statistical metrics. For example, an error log reportedby a load balancer (e.g., the load balancer 510-1, FIG. 5) may be:“Error(s) found in configuration file: letclhaproxy/haproxy.cfg” and anerror reported by an application server (e.g., one of the applicationservers 520, FIG. 5) may be: “Connection pool is full. discardingconnection: [ConnectionName], path name: [PathName]”. Thus, the sever130 is configured to determine the root-cause of any incidence by merelyprocessing input machine-generated textual data.

It should be understood that the embodiments disclosed herein are notlimited to the specific architecture illustrated in FIG. 1, and otherarchitectures may be equally used without departing from the scope ofthe disclosed embodiments. Specifically, the server 130 may reside in acloud computing platform, a datacenter, and the like. Moreover, in anembodiment, there may be a plurality of classification servers operatingas described hereinabove and configured to either have one as a standby,to share the load between them, or to split the functions between them.

FIG. 6 shows an example flowchart 600 of a method for determining aroot-cause of an incidence based machine-generated textual dataaccording to an embodiment. At S610, machine-generated textual data isreceived from a plurality of data sources. The machine-generated textualdata may include, but is not limited to, application logs, configurationfiles, messages, spreadsheets, alerts, sensory signals, audit records,combinations thereof, and the like.

At S620, the received machine-generated textual data is classified intostatistical metrics. As noted above, a statistical metric may include,but is not limited, a gauge, a meter, a histogram, and the like. In anembodiment, the classification process is performed as described in moredetail in the above-referenced '272 Application.

At S630, the statistical metrics are processed to recognize incidencepatterns. In an embodiment, S630 includes applying a past behavior ofeach incidence patterns to predict its future behavior. The incidencepatterns may include, but are not limited to, new behavior, anomalousbehavior, changes in routine operational, ongoing trends, and so on. Asnoted above, some techniques for recognizing incidence patterns that canbe utilized according to the disclosed embodiments are discussed above.

At S640, a set of meaningful incidence patterns are selected orotherwise determined from among the recognized incidence patterns. Ameaningful incidence pattern is an incidence pattern containing data (ofa statistical metric) that can be utilized to determine a root-cause ofan observed or unobserved incidence. Such meaningful patterns can bedetected by considering the amplitude of a pattern, a frequency of apattern, a similarity of a pattern to previously detected patterns, anumber of detected same or similar patterns, or any combination thereof.For example, only incidence patterns that were previously observed areconsidered meaningful. In some embodiments, S640 is optional.

At S650, the determined meaningful incidence patterns are analyzed toidentify any root-cause of an incidence (observed or unobserved). Anobserved incidence may be reported in an alert, while an unobservedincidence exists in the monitored environment but has not beendiscovered yet. In an embodiment, the monitored environment is any ITinfrastructure and any computing resource included therein. Examples forsuch resources include network devices, computers, servers, softwareapplications, software services, storage devices, storage networks, andthe like. An incidence may be, for example, a failure or malfunctionthat negatively affects the operation of such a resource.

In an embodiment, the analysis includes correlating the incidencepatterns (meaningful or not) using different correlation types. Such acorrelation type may include time-proximity correlation, component-based(i.e., correlation of incidence patterns across different components),order-based (i.e., correlation of incidence patterns based on a certainorder), or a combination thereof. The root-cause is determined based onthe correlation type being utilized. The different correlation types arediscussed in greater detail below and for the sake of the simplicity ofthe discussion are not repeated herein.

At S660, it is checked if a root-cause of an incidence is determinedbased on the analysis of the meaningful incidence patterns. If so,execution continues with S670, where an alert is generated and reported;otherwise, execution proceeds with S680. The generated alert indicatesat least the determined root-cause. In another embodiment, any alertremaining after the root-cause analysis with additional informationrelated to the associated incidence or incidences is also reported. AtS680, it is checked if there are additional data logs to be processed;if so, execution returns to S610; otherwise, execution terminates,

FIG. 7 shows an example block diagram of the server 130 implementedaccording to an embodiment. The server 130 includes a processing circuit710 coupled to a memory 715, a storage 720, a classifier 730, and anetwork interface 740. In an embodiment, the components of theclassification server 130 may be communicatively connected via a bus760.

The processing circuit 710 may be realized as one or more hardware logiccomponents and circuits. For example, and without limitation,illustrative types of hardware logic components that can be used includefield programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), Application-specific standard products (ASSPs),system-on-a-chip systems (SOCs), general-purpose microprocessors,microcontrollers, digital signal processors (DSPs), and the like, or anyother hardware logic components that can perform calculations or othermanipulations of information.

The memory 715 may be volatile (e.g., RAM, etc.), non-volatile (e.g.,ROM, flash memory, etc.), or a combination thereof. In oneconfiguration, computer readable instructions to implement one or moreembodiments disclosed herein may be stored in the storage 720.

In another embodiment, the memory 715 is configured to store software.Software shall be construed broadly to mean any type of instructions,whether referred to as software, firmware, middleware, microcode,hardware description language, or otherwise. Instructions may includecode (e.g., in source code format, binary code format, executable codeformat, or any other suitable format of code). The instructions, whenexecuted by the one or more processors, cause the processing circuit 710to perform the various processes described herein. Specifically, theinstructions, when executed, configure the processing circuit 710 todetermine root-cause of incidences based on machine-generated textualdata. In a further embodiment, the memory 715 may further include amemory portion 717 including the instructions.

The storage 720 may be magnetic storage, optical storage, and the like,and may be realized, for example, as flash memory or other memorytechnology, CD-ROM, Digital Versatile Disks (DVDs), or any other mediumwhich can be used to store the desired information. The storage 720 maystore the received machine-generated textual data, events, metadata forevents, events schema, various elements of each events, and/or theclassification results.

The classifier 730 is configured to classify machine-generated textualdata into metrics to enable systems to ingest, process, analyze,aggregate, and correlate the data by machines without scale or volumelimitations. In an embodiment, processes performed by the classifier 730may be performed as discussed in greater detail above, at least withrespect to FIG. 6.

The network interface 740 allows the server 130 to communicate with thedata sources 140 for the purpose of, for example, receivingmachine-generated textual data. The server 130 can be utilized to outputthe classification results to external systems (not shown) for furtherprocessing. In an embodiment, the network interface 740 can allowinterface with client devices to view the classification results and/orto configure the server 130.

It should be understood that the embodiments described herein are notlimited to the specific architecture illustrated in FIG. 7, and thatother architectures may be equally used without departing from the scopeof the disclosed embodiments.

The various embodiments disclosed herein can be implemented as hardware,firmware, software, or any combination thereof. Moreover, the softwareis preferably implemented as an application program tangibly embodied ona program storage unit or computer readable medium consisting of parts,or of certain devices and/or a combination of devices. The applicationprogram may be uploaded to, and executed by, a machine comprising anysuitable architecture. Preferably, the machine is implemented on acomputer platform having hardware such as one or more central processingunits (“CPUs”), a memory, and input/output interfaces. The computerplatform may also include an operating system and microinstruction code.The various processes and functions described herein may be either partof the microinstruction code or part of the application program, or anycombination thereof, which may be executed by a CPU, whether or not sucha computer or processor is explicitly shown. In addition, various otherperipheral units may be connected to the computer platform such as anadditional data storage unit and a printing unit. Furthermore, anon-transitory computer readable medium is any computer readable mediumexcept for a transitory propagating signal.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the principlesof the disclosed embodiment and the concepts contributed by the inventorto furthering the art, and are to be construed as being withoutlimitation to such specifically recited examples and conditions.Moreover, all statements herein reciting principles, aspects, andembodiments of the disclosed embodiments, as well as specific examplesthereof, are intended to encompass both structural and functionalequivalents thereof. Additionally, it is intended that such equivalentsinclude both currently known equivalents as well as equivalentsdeveloped in the future, i.e., any elements developed that perform thesame function, regardless of structure.

What is claimed is:
 1. A system, comprising: a processor; and a memory,accessible by the processor, the memory storing instructions, that whenexecuted by the processor, cause the processor to perform operationscomprising: receiving machine-generated textual data associated with oneor more computing resources of a managed network; identifying aplurality of incident patterns associated with respective incidents inthe managed network based on the machine-generated textual data;correlating at least two incident patterns of the plurality of incidentpatterns; determining a root cause of the respective incidentsassociated with the at least two incident patterns based on an increasein a trend of one of the at least two incident patterns toward athreshold, or a time-proximity between respective anomalies in the atleast two incident patterns, or both; and generating an alert for therespective incidents associated with the at least two incident patterns,wherein the alert indicates the determined root cause.
 2. The system ofclaim 1, wherein identifying a plurality of incident patterns comprises:generating one or more statistical metrics based on themachine-generated textual data; and identifying the plurality ofincident patterns based on the one or more statistical metrics.
 3. Thesystem of claim 2, wherein the one or more statistical metrics arerepresented as a gauge, a meter, or a histogram, or a combinationthereof.
 4. The system of claim 2, wherein generating the one or morestatistical metrics comprises: processing the machine-generated textualdata into one or more elements including one or more events, one or moretokens, one or more key-value pairs, or one or more properties, or acombination thereof; and generating graphical representations of the oneor more statistical metrics based on the one or more elements.
 5. Thesystem of claim 3, wherein the alert comprises the one or more elements.6. The system of claim 1, wherein the at least two incident patterns arecorrelated based on an amplitude of an incident pattern of the at leasttwo incident patterns, a frequency of an incident pattern of the atleast two incident patterns, or a similarity of an incident pattern ofthe at least two incident patterns to a previously detected incidentpattern, or a combination thereof.
 7. The system of claim 1, wherein theoperations comprise grouping the alert with one or more additionalalerts generated at the same time or substantially at the same time. 8.A method, comprising: receiving, by one or more processors,machine-generated textual data associated with one or more computingresources of a managed network; generating, by the one or moreprocessors, one or more statistical metrics based on themachine-generated textual data; identifying, by the one or moreprocessors, a plurality of incident patterns associated with respectiveincidents in the managed network based on the one or more statisticalmetrics; correlating, by the one or more processors, at least twoincident patterns of the plurality of incident patterns; determining, bythe one or more processors, a root cause of the respective incidentsassociated with the at least two incident patterns based on an increasein a trend of one of the at least two incident patterns toward athreshold, or a time-proximity between respective anomalies in the atleast two incident patterns, or both; and generating, by the one or moreprocessors, an alert for the respective incidents associated with the atleast two incident patterns, wherein the alert indicates the determinedroot cause.
 9. The method of claim 8, wherein the one or morestatistical metrics are represented as a gauge, a meter, or a histogram,or a combination thereof.
 10. The method of claim 8, wherein generatingthe one or more statistical metrics comprises: processing, by the one ormore processors, the machine-generated textual data into one or moreelements including one or more events, one or more tokens, one or morekey-value pairs, or one or more properties, or a combination thereof;and generating, by the one or more processors, graphical representationsof the one or more statistical metrics based on the one or moreelements.
 11. The method of claim 9, wherein the alert comprises the oneor more elements.
 12. The method of claim 8, wherein the at least twoincident patterns are correlated based on an amplitude of an incidentpattern of the at least two incident patterns, a frequency of anincident pattern of the at least two incident patterns, or a similarityof an incident pattern of the at least two incident patterns to apreviously detected incident pattern, or a combination thereof.
 13. Themethod of claim 8, comprising grouping the alert with one or moreadditional alerts generated at the same time or substantially at thesame time.
 14. The method of claim 8, wherein the respective incidentsassociated with the at least two incident patterns comprise anunobserved incident in the managed network.
 15. A non-transitory,computer-readable medium, comprising instructions that when executed byone or more processors, cause the one or more processors to performoperations comprising: receiving machine-generated textual dataassociated with one or more computing resources of a managed network;processing the machine-generated textual data into one or more elements;generating one or more statistical metrics based on the one or moreelements; identifying a plurality of incident patterns associated withrespective incidents in the managed network based on the one or morestatistical metrics; correlating at least two incident patterns of theplurality of incident patterns; determining a root cause of therespective incidents associated with the at least two incident patternsbased on an increase in a trend of one of the at least two incidentpatterns toward a threshold, or a time-proximity between respectiveanomalies in the at least two incident patterns, or both; and generatingan alert for the respective incidents associated with the at least twoincident patterns, wherein the alert indicates the determined root causeand at least one of the one or more elements.
 16. The non-transitory,computer-readable medium of claim 15, wherein the one or morestatistical metrics are represented as a gauge, a meter, or a histogram,or a combination thereof.
 17. The non-transitory, computer-readablemedium of claim 15, wherein the one or more elements comprise one ormore events, one or more tokens, one or more key-value pairs, or one ormore properties, or a combination thereof.
 18. The non-transitory,computer-readable medium of claim 15, wherein the at least two incidentpatterns are correlated based on an amplitude of an incident pattern ofthe at least two incident patterns, a frequency of an incident patternof the at least two incident patterns, or a similarity of an incidentpattern of the at least two incident patterns to a previously detectedincident pattern, or a combination thereof.
 19. The non-transitory,computer-readable medium of claim 15, wherein the operations comprisegrouping the alert with one or more additional alerts generated at thesame time or substantially at the same time.
 20. The non-transitory,computer-readable medium of claim 15, wherein the respective incidentsassociated with the at least two incident patterns comprise anunobserved incident in the managed network.